San Francisco Housing Rental Analysis

In this assignment, you will perform basic analysis for the San Francisco Housing Market to allow potential real estate investors to choose rental investment properties.

In [1]:
# initial imports
import os
import pandas as pd
import matplotlib.pyplot as plt
import hvplot.pandas
import plotly.express as px
from pathlib import Path
from dotenv import load_dotenv

%matplotlib inline

# Add additional imports
import numpy as np
import panel as pn
from panel.interact import interact
from panel import widgets
In [2]:
# Read the Mapbox API key
load_dotenv()
# mapbox_token = os.getenv("MAPBOX")
mapbox_token = os.getenv("MAPBOX_API_KEY")
# the .env file with key will be removed from local folder before uploading to Github Repo

Load Data

In [3]:
# Read the census data into a Pandas DataFrame
file_path = Path("Data/sfo_neighborhoods_census_data.csv")
sfo_data = pd.read_csv(file_path, index_col="year")
sfo_data.head()
Out[3]:
neighborhood sale_price_sqr_foot housing_units gross_rent
year
2010 Alamo Square 291.182945 372560 1239
2010 Anza Vista 267.932583 372560 1239
2010 Bayview 170.098665 372560 1239
2010 Buena Vista Park 347.394919 372560 1239
2010 Central Richmond 319.027623 372560 1239

Housing Units Per Year

In this section, you will calculate the number of housing units per year and visualize the results as a bar chart using the Pandas plot function.

Hint: Use the Pandas groupby function

Optional challenge: Use the min, max, and std to scale the y limits of the chart.

In [4]:
# Calculate the mean number of housing units per year (hint: use groupby) 
# YOUR CODE HERE!

# First group by year, then drop columns that are not requested to show
sfo_data_grp1 = sfo_data.groupby('year').mean()
sfo_data_grp2 = sfo_data_grp1.drop(columns=['sale_price_sqr_foot','gross_rent'])
sfo_data_grp2.head 
Out[4]:
<bound method NDFrame.head of       housing_units
year               
2010         372560
2011         374507
2012         376454
2013         378401
2014         380348
2015         382295
2016         384242>
In [5]:
# Use the Pandas plot function to plot the average housing units per year.
# Note: You will need to manually adjust the y limit of the chart using the min and max values from above.
# YOUR CODE HERE!

sfo_data_grp2.hvplot.bar(label='Housing Units per year', x='year',y='housing_units', xlabel='Year', ylabel='Housing Units')

# Optional Challenge: Use the min, max, and std to scale the y limits of the chart
# YOUR CODE HERE!

# plt.show()
# plt.close(fig_housing_units)
Out[5]:

Average Prices per Square Foot

In this section, you will calculate the average gross rent and average sales price for each year. Plot the results as a line chart.

Average Gross Rent in San Francisco Per Year

In [6]:
# Calculate the average gross rent and average sale price per square foot
# YOUR CODE HERE!
sfo_data_avg1=sfo_data
sfo_data_avg1.head()

# First group by year, then drop columns that are not requested to show
sfo_data_grp11 = sfo_data.groupby('year').mean()
sfo_data_grp12 = sfo_data_grp1.drop(columns=['housing_units'])
sfo_data_grp12.head
Out[6]:
<bound method NDFrame.head of       sale_price_sqr_foot  gross_rent
year                                 
2010           369.344353        1239
2011           341.903429        1530
2012           399.389968        2324
2013           483.600304        2971
2014           556.277273        3528
2015           632.540352        3739
2016           697.643709        4390>
In [7]:
# Plot the Average Gross Rent per Year as a Line Chart 
# YOUR CODE HERE!
sfo_data_grp13 = sfo_data_grp12.drop(columns=['sale_price_sqr_foot'])
sfo_data_grp13.plot(title='Average Gross Rent in San Fransisco',legend=False)
Out[7]:
<matplotlib.axes._subplots.AxesSubplot at 0x1af15bd7bc8>

Average Sales Price per Year

In [8]:
# Plot the Average Sales Price per Year as a line chart
# YOUR CODE HERE!
sfo_data_grp14 = sfo_data_grp12.drop(columns=['gross_rent'])
sfo_data_grp14.plot(title='Average Sale Price per Square Foot in San Fransisco',legend=False)
Out[8]:
<matplotlib.axes._subplots.AxesSubplot at 0x1af16327948>

Average Prices by Neighborhood

In this section, you will use hvplot to create an interactive visulization of the Average Prices with a dropdown selector for the neighborhood.

Hint: It will be easier to create a new DataFrame from grouping the data and calculating the mean prices for each year and neighborhood

In [9]:
# Group by year and neighborhood and then create a new dataframe of the mean values
# YOUR CODE HERE!
# First group by year, then drop columns that are not requested to show
# sfo_data_grp21 = sfo_data.groupby(['year','neighborhood']).mean()
sfo_data_grp21 = sfo_data
sfo_data_grp21.reset_index(inplace=True)
sfo_data_grp21.head(10)
Out[9]:
year neighborhood sale_price_sqr_foot housing_units gross_rent
0 2010 Alamo Square 291.182945 372560 1239
1 2010 Anza Vista 267.932583 372560 1239
2 2010 Bayview 170.098665 372560 1239
3 2010 Buena Vista Park 347.394919 372560 1239
4 2010 Central Richmond 319.027623 372560 1239
5 2010 Central Sunset 418.172493 372560 1239
6 2010 Corona Heights 369.359338 372560 1239
7 2010 Cow Hollow 569.379968 372560 1239
8 2010 Croker Amazon 165.645730 372560 1239
9 2010 Diamond Heights 456.930822 372560 1239
In [10]:
# Use hvplot to create an interactive line chart of the average price per sq ft.
# The plot should have a dropdown selector for the neighborhood
# YOUR CODE HERE!
# Slice data
sfo_data_grp22 = sfo_data_grp21.drop(columns=['housing_units','gross_rent'])
sfo_data_grp22.groupby(['year','neighborhood']).mean().sort_values('year')
sfo_data_grp22.head
sfo_data_grp22.hvplot('year','sale_price_sqr_foot')
Out[10]:

The Top 10 Most Expensive Neighborhoods

In this section, you will need to calculate the mean sale price for each neighborhood and then sort the values to obtain the top 10 most expensive neighborhoods on average. Plot the results as a bar chart.

In [11]:
# Getting the data from the top 10 expensive neighborhoods
# YOUR CODE HERE!
# First Sort and then group-by
sfo_data_grp31 = sfo_data
# sfo_data_grp31.reset_index(inplace=True)
sfo_data_grp32 = sfo_data_grp31.drop(columns=['year'])
sfo_data_grp33 = sfo_data_grp32.groupby('neighborhood').mean().sort_values('sale_price_sqr_foot',ascending=False)
sfo_data_grp33.head(10)
Out[11]:
sale_price_sqr_foot housing_units gross_rent
neighborhood
Union Square District 903.993258 377427.50 2555.166667
Merced Heights 788.844818 380348.00 3414.000000
Miraloma Park 779.810842 375967.25 2155.250000
Pacific Heights 689.555817 378401.00 2817.285714
Westwood Park 687.087575 382295.00 3959.000000
Telegraph Hill 676.506578 378401.00 2817.285714
Presidio Heights 675.350212 378401.00 2817.285714
Cow Hollow 665.964042 378401.00 2817.285714
Potrero Hill 662.013613 378401.00 2817.285714
South Beach 650.124479 375805.00 2099.000000
In [12]:
# Plotting the data from the top 10 expensive neighborhoods
# YOUR CODE HERE!
# Create a DF with Top 10 Records
sfo_data_grp34 = sfo_data_grp33.iloc[0:10]
# Create a DF by dropping columns
sfo_data_grp35 = sfo_data_grp34.drop(columns=['housing_units','gross_rent'])
sfo_data_grp35.hvplot.bar(label='Top 10 Expensive Neighborhoods', x='neighborhood',y='sale_price_sqr_foot', xlabel='Neighborhood', ylabel='Sales price Sqr Foot')
Out[12]:

Parallel Coordinates and Parallel Categories Analysis

In this section, you will use plotly express to create parallel coordinates and parallel categories visualizations so that investors can interactively filter and explore various factors related to the sales price of the neighborhoods.

Using the DataFrame of Average values per neighborhood (calculated above), create the following visualizations:

  1. Create a Parallel Coordinates Plot
  2. Create a Parallel Categories Plot
In [13]:
# Parallel Coordinates Plot
# YOUR CODE HERE!
px.parallel_coordinates(sfo_data_grp34)
In [14]:
# Parallel Categories Plot
# YOUR CODE HERE!
sfo_data_grp41 = sfo_data_grp34
sfo_data_grp41.reset_index(inplace=True)
px.parallel_categories(sfo_data_grp41)

Neighborhood Map

In this section, you will read in neighboor location data and build an interactive map with the average prices per neighborhood. Use a scatter_mapbox from plotly express to create the visualization. Remember, you will need your mapbox api key for this.

Load Location Data

In [15]:
# Load neighborhoods coordinates data
file_path = Path("Data/neighborhoods_coordinates.csv")
df_neighborhood_locations = pd.read_csv(file_path)
df_neighborhood_locations.head()
Out[15]:
Neighborhood Lat Lon
0 Alamo Square 37.791012 -122.402100
1 Anza Vista 37.779598 -122.443451
2 Bayview 37.734670 -122.401060
3 Bayview Heights 37.728740 -122.410980
4 Bernal Heights 37.728630 -122.443050

Data Preparation

You will need to join the location data with the mean prices per neighborhood

  1. Calculate the mean values for each neighborhood
  2. Join the average values with the neighborhood locations
In [16]:
# Calculate the mean values for each neighborhood
# YOUR CODE HERE!
# Drop column - Year, Group By Neighborhood, then reset-index
sfo_data_grp51 = sfo_data.drop(columns=['year']).groupby('neighborhood').mean()
sfo_data_grp51.reset_index(inplace=True)
sfo_data_grp51.head()
Out[16]:
neighborhood sale_price_sqr_foot housing_units gross_rent
0 Alamo Square 366.020712 378401.0 2817.285714
1 Anza Vista 373.382198 379050.0 3031.833333
2 Bayview 204.588623 376454.0 2318.400000
3 Bayview Heights 590.792839 382295.0 3739.000000
4 Bernal Heights 576.746488 379374.5 3080.333333
In [17]:
# Join the average values with the neighborhood locations
# YOUR CODE HERE!
df_neighborhood_locations_11 = df_neighborhood_locations.set_index('Neighborhood')
sfo_data_grp52 = sfo_data_grp51.set_index('neighborhood')
merged_data = pd.concat([df_neighborhood_locations_11, sfo_data_grp52],axis='columns', join = 'inner')
merged_data_11=merged_data.reset_index()
merged_data_12=merged_data_11.rename(columns={'index': 'Neighborhood'})
merged_data_12.head()
Out[17]:
Neighborhood Lat Lon sale_price_sqr_foot housing_units gross_rent
0 Alamo Square 37.791012 -122.402100 366.020712 378401.0 2817.285714
1 Anza Vista 37.779598 -122.443451 373.382198 379050.0 3031.833333
2 Bayview 37.734670 -122.401060 204.588623 376454.0 2318.400000
3 Bayview Heights 37.728740 -122.410980 590.792839 382295.0 3739.000000
4 Buena Vista Park 37.768160 -122.439330 452.680591 378076.5 2698.833333

Mapbox Visualization

Plot the aveage values per neighborhood with a plotly express scatter_mapbox visualization.

In [18]:
# Create a scatter mapbox to analyze neighborhood info
# YOUR CODE HERE!

# Set the Mapbox API
px.set_mapbox_access_token(mapbox_token)
map = px.scatter_mapbox(
    merged_data_12,
    title='Average Sales Price per Square Foot and Gross Rent in San Francisco',
    lat='Lat',
    lon='Lon',
    size='sale_price_sqr_foot',
    color='gross_rent',
    hover_name='Neighborhood',
    zoom=11
)
map.show()
In [ ]: